Frog Game Generalized Mixed Modeling
JMG
2024-07-12
data <- read_csv("/Users/jorgevargasmutizabal/Desktop/Frog game statistics/R Analysis Frog/IN/nat.csv")## Rows: 22404 Columns: 8
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## dbl (7): Player_ID, LexTale, Game_Version, Game_Level, Phrase_Condition, Que...
## lgl (1): Answer
##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.str(data)## spc_tbl_ [22,404 × 8] (S3: spec_tbl_df/tbl_df/tbl/data.frame)
## $ Player_ID : num [1:22404] 1 1 1 1 1 1 1 1 1 1 ...
## $ LexTale : num [1:22404] 8 8 8 8 8 8 8 8 8 8 ...
## $ Game_Version : num [1:22404] 1 1 1 1 1 1 1 1 1 1 ...
## $ Game_Level : num [1:22404] 1 1 1 1 1 1 1 1 1 1 ...
## $ Phrase_Condition: num [1:22404] 1 1 1 1 1 1 1 1 1 1 ...
## $ Question_Num : num [1:22404] 0 1 2 3 4 5 6 7 8 9 ...
## $ Answer : logi [1:22404] TRUE TRUE TRUE TRUE TRUE TRUE ...
## $ Reaction_Time : num [1:22404] 1.99 0.8 1.8 0.77 0.82 0.86 0.82 0.85 0.78 1.06 ...
## - attr(*, "spec")=
## .. cols(
## .. Player_ID = col_double(),
## .. LexTale = col_double(),
## .. Game_Version = col_double(),
## .. Game_Level = col_double(),
## .. Phrase_Condition = col_double(),
## .. Question_Num = col_double(),
## .. Answer = col_logical(),
## .. Reaction_Time = col_double()
## .. )
## - attr(*, "problems")=<externalptr>data <- data %>%
mutate(
Player_ID = as.factor(Player_ID), # Convert to factor
LexTale = as.numeric(LexTale), # Convert to numeric
Game_Version = as.factor(Game_Version), # Convert to factor
Game_Level = as.factor(Game_Level), # Convert to factor
Phrase_Condition = as.factor(Phrase_Condition), # Convert to factor
Question_Num = as.numeric(Question_Num), # Convert to numeric
Answer = as.numeric(Answer), # Convert to factor
Reaction_Time = as.numeric(Reaction_Time) # Convert to numeric
)
# Verify changes
str(data)## tibble [22,404 × 8] (S3: tbl_df/tbl/data.frame)
## $ Player_ID : Factor w/ 38 levels "1","2","3","4",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ LexTale : num [1:22404] 8 8 8 8 8 8 8 8 8 8 ...
## $ Game_Version : Factor w/ 4 levels "1","2","3","4": 1 1 1 1 1 1 1 1 1 1 ...
## $ Game_Level : Factor w/ 8 levels "1","2","3","4",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ Phrase_Condition: Factor w/ 4 levels "1","2","3","4": 1 1 1 1 1 1 1 1 1 1 ...
## $ Question_Num : num [1:22404] 0 1 2 3 4 5 6 7 8 9 ...
## $ Answer : num [1:22404] 1 1 1 1 1 1 1 1 0 1 ...
## $ Reaction_Time : num [1:22404] 1.99 0.8 1.8 0.77 0.82 0.86 0.82 0.85 0.78 1.06 ...Model 1: Simple Random Intercept
Formula:
Reaction_Time ~ 1 + Question_Num + Phrase_Condition * Game_Version + (1 | Player_ID)
Can Answer: - How do reaction times generally relate
to Question_Num, Phrase_Condition, and
Game_Version? - Are there overall differences in reaction
times due to the interaction between Phrase_Condition and
Game_Version? - What is the baseline variation in reaction
times across different players?
Cannot Answer: - Does the effect of progressing through questions vary among individual players? - Do individual players react differently to various phrase conditions? - How do individual differences affect the interaction between phrase conditions and game versions?
Model 2: Adding Random Slope for Question Number
Formula:
Reaction_Time ~ 1 + Question_Num + Phrase_Condition * Game_Version + (1 + Question_Num | Player_ID)
Can Answer: - All questions Model 1 can answer. -
How does the impact of Question_Num on reaction times vary
across individual players?
Cannot Answer: - Do different players have distinct reactions to the various phrase conditions, beyond just a baseline reaction time difference? - How do individual differences interact with the combined effects of phrase conditions and game versions?
Model 3: Adding Random Slope for Phrase Condition
Formula:
Reaction_Time ~ 1 + Question_Num + Phrase_Condition * Game_Version + (1 + Phrase_Condition | Player_ID)
Can Answer: - All questions Model 1 can answer. -
How does the impact of different Phrase_Conditions on
reaction times vary among individual players?
Cannot Answer: - How do individual differences in reaction to questions over time affect reaction times? - Does the relationship between phrase condition and game version differ by player, considering their specific reaction to both conditions and questions over time?
Model 4: Combining Random Slopes for Both Question Number and Phrase Condition
Formula:
Reaction_Time ~ 1 + Question_Num + Phrase_Condition * Game_Version + (1 + Question_Num + Phrase_Condition | Player_ID)
Can Answer: - All questions that Models 1, 2, and 3
can answer. - How do individual differences in reaction to both
Question_Num and Phrase_Condition interact to
affect reaction times? - Does the way a player’s reaction time changes
with questions relate to how they react to different phrase
conditions?
Cannot Answer: - Specific inquiries about
third-level interactions involving game version beyond the fixed effects
structure, unless further random slopes for Game_Version or
its interaction terms are included. - Questions about more complex
nested or crossed random effects not specified in the model (e.g.,
random slopes for Game_Version, or cross-level interactions
such as Phrase_Condition:Game_Version | Player_ID).
Summary
Each model’s ability to answer specific questions is tied to the random effects it includes. As you add more complexity to the model (like in Model 4), you can address more detailed and specific questions about individual variations and interactions between the effects. However, each addition also requires more data and computational power, and the risk of overfitting or convergence issues increases. Therefore, selecting a model should balance the need for detail with the practicality of fitting the model and the quality of the data available.
Let’s fit some models!
0. What is the average reaction time marginalizing over everything?
Oa: rt ~ 1 + (1 | participant)
Fitting this model allows you to determine how much variation in reaction times is due to individual differences between players versus random noise or other factors. A significant Player_ID variance component suggests individual differences are important; if it’s close to zero, reaction times are consistent across players, and the random effect may be unnecessary.
model1 <- lmer(Reaction_Time ~ 1 + (1 | Player_ID), data = data)
# Obtain and print a summary of the model to examine fixed and random effects
print(summary(model1))## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: Reaction_Time ~ 1 + (1 | Player_ID)
## Data: data
##
## REML criterion at convergence: 64205.2
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -1.470 -0.389 -0.111 0.200 46.838
##
## Random effects:
## Groups Name Variance Std.Dev.
## Player_ID (Intercept) 0.04523 0.2127
## Residual 1.02245 1.0112
## Number of obs: 22404, groups: Player_ID, 38
##
## Fixed effects:
## Estimate Std. Error df t value Pr(>|t|)
## (Intercept) 0.9557 0.0352 37.1878 27.15 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1#Diagnostic plots
plot_residuals_diagnostics(model1)
## `geom_smooth()` using formula = 'y ~ x'
# Model diagnostics
check_model(model1)
Random Effects Player_ID: The variance for random intercepts attributed to Player_ID is 0.01319 with a standard deviation of 0.1149, indicating small variability in reaction times among players compared to residual variability.
Residual Variance: The residual variance is 0.16394 with a standard deviation of 0.4049, showing that most variability in reaction times is due to unaccounted factors or inherent variability in individual reactions.
Fixed Effects Intercept (Overall Average Reaction Time): The intercept estimate is 0.83745 with a standard error of 0.01885, indicating an average reaction time of approximately 0.837 seconds. The t-value of 44.43 and a p-value of less than 0.001 confirm that this average reaction time is significantly different from zero.
Interpretation of the Model: The analysis shows small variability in reaction times due to individual differences (Player_ID) with most variance captured by the residual term. This suggests either consistent game difficulty across players or minimal variation in individual reaction times.
1. Does learning take place?
1b: rt ~ 1 + time + (1 + time | participant)
model2 <- lmer(Reaction_Time ~ 1 + Question_Num + (1 + Question_Num | Player_ID), data = data)## Warning in checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, :
## Model failed to converge with max|grad| = 78.1395 (tol = 0.002, component 1)## Warning in checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, : Model is nearly unidentifiable: very large eigenvalue
## - Rescale variables?;Model is nearly unidentifiable: large eigenvalue ratio
## - Rescale variables?# Obtain and print a summary of the model to examine fixed and random effects
print(summary(model2))## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: Reaction_Time ~ 1 + Question_Num + (1 + Question_Num | Player_ID)
## Data: data
##
## REML criterion at convergence: 63625.1
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -2.506 -0.381 -0.101 0.209 47.588
##
## Random effects:
## Groups Name Variance Std.Dev. Corr
## Player_ID (Intercept) 3.328e-01 0.576871
## Question_Num 3.768e-06 0.001941 -0.95
## Residual 9.907e-01 0.995346
## Number of obs: 22404, groups: Player_ID, 38
##
## Fixed effects:
## Estimate Std. Error df t value Pr(>|t|)
## (Intercept) 1.0787300 0.0945331 7.1674459 11.411 7.44e-06 ***
## Question_Num -0.0007861 0.0003216 8.4820061 -2.445 0.0386 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr)
## Question_Nm -0.947
## optimizer (nloptwrap) convergence code: 0 (OK)
## Model failed to converge with max|grad| = 78.1395 (tol = 0.002, component 1)
## Model is nearly unidentifiable: very large eigenvalue
## - Rescale variables?
## Model is nearly unidentifiable: large eigenvalue ratio
## - Rescale variables?#Diagnostic plots
plot_residuals_diagnostics(model2)
## `geom_smooth()` using formula = 'y ~ x'
# Diagnostic checks - Simulating residuals to validate model assumptions
residuals_simulation <- simulateResiduals(fittedModel = model2, n = 500)
plot(residuals_simulation)
# Model diagnostics
check_model(model2)
# Check for influential cases potentially affecting the model
#influence_measures <- influence(model3)
#plot(influence_measures, which = "cook")
# If assumptions are not met, consider re-fitting the model with transformations or different specifications
# model_revised <- update(model, . ~ . + log(Reaction_Time))
# print(summary(model_revised))
# Model interpretation - examining fixed and random effects
#fixed_effects <- fixef(model3)
#random_effects <- ranef(model3)
# Printing fixed effects for interpretation
#print(fixed_effects)
# Printing random effects for interpretation
#print(random_effects)
# Get estimated marginal means for specific values of Question_Num
emm_results <- emmeans(model2, specs = ~ Question_Num, at = list(Question_Num = seq(min(data$Question_Num), max(data$Question_Num), by = 10)))## Note: D.f. calculations have been disabled because the number of observations exceeds 3000.
## To enable adjustments, add the argument 'pbkrtest.limit = 22404' (or larger)
## [or, globally, 'set emm_options(pbkrtest.limit = 22404)' or larger];
## but be warned that this may result in large computation time and memory use.
## Note: D.f. calculations have been disabled because the number of observations exceeds 3000.
## To enable adjustments, add the argument 'lmerTest.limit = 22404' (or larger)
## [or, globally, 'set emm_options(lmerTest.limit = 22404)' or larger];
## but be warned that this may result in large computation time and memory use.# Summarize the results
summary(emm_results)## Question_Num emmean SE df asymp.LCL asymp.UCL
## 0 1.079 0.0945 Inf 0.893 1.264
## 10 1.071 0.0915 Inf 0.892 1.250
## 20 1.063 0.0885 Inf 0.890 1.236
## 30 1.055 0.0855 Inf 0.888 1.223
## 40 1.047 0.0825 Inf 0.886 1.209
## 50 1.039 0.0795 Inf 0.884 1.195
## 60 1.032 0.0765 Inf 0.882 1.182
## 70 1.024 0.0736 Inf 0.879 1.168
## 80 1.016 0.0707 Inf 0.877 1.154
## 90 1.008 0.0678 Inf 0.875 1.141
## 100 1.000 0.0649 Inf 0.873 1.127
## 110 0.992 0.0621 Inf 0.871 1.114
## 120 0.984 0.0593 Inf 0.868 1.101
## 130 0.977 0.0566 Inf 0.866 1.087
## 140 0.969 0.0539 Inf 0.863 1.074
## 150 0.961 0.0513 Inf 0.860 1.061
## 160 0.953 0.0487 Inf 0.857 1.048
## 170 0.945 0.0463 Inf 0.854 1.036
## 180 0.937 0.0439 Inf 0.851 1.023
## 190 0.929 0.0417 Inf 0.848 1.011
## 200 0.922 0.0395 Inf 0.844 0.999
## 210 0.914 0.0376 Inf 0.840 0.987
## 220 0.906 0.0358 Inf 0.836 0.976
## 230 0.898 0.0342 Inf 0.831 0.965
## 240 0.890 0.0329 Inf 0.826 0.955
## 250 0.882 0.0318 Inf 0.820 0.945
## 260 0.874 0.0311 Inf 0.813 0.935
## 270 0.866 0.0306 Inf 0.806 0.926
## 280 0.859 0.0305 Inf 0.799 0.918
## 290 0.851 0.0307 Inf 0.791 0.911
## 300 0.843 0.0313 Inf 0.782 0.904
## 310 0.835 0.0322 Inf 0.772 0.898
## 320 0.827 0.0333 Inf 0.762 0.893
## 330 0.819 0.0348 Inf 0.751 0.887
## 340 0.811 0.0364 Inf 0.740 0.883
## 350 0.804 0.0383 Inf 0.729 0.879
## 360 0.796 0.0403 Inf 0.717 0.875
## 370 0.788 0.0424 Inf 0.705 0.871
## 380 0.780 0.0447 Inf 0.692 0.868
##
## Degrees-of-freedom method: asymptotic
## Confidence level used: 0.95# Plot the estimated marginal means
plot(emm_results)
# Create a plot of Reaction_Time vs. Question_Num with a regression line
ggplot(data, aes(x = Question_Num, y = Reaction_Time)) +
geom_point(alpha = 0.3) +
geom_smooth(method = "lm", se = TRUE, color = "blue") +
labs(title = "Reaction Time vs. Question Number",
x = "Question Number",
y = "Reaction Time")## `geom_smooth()` using formula = 'y ~ x'
2. Which condition is the hardest overall?
2a: rt - 1 + condition + (1 + condition | participant)
1 | Player_ID: Introduces a random intercept for each player, allowing baseline reaction times to vary. Phrase_Condition | Player_ID: Adds random slopes for the effect of Phrase_Condition on reaction time for each player, indicating that players may react differently to each phrase condition. Implications of Including Random Slopes: Individual Variability: Acknowledges that the effect of phrase conditions varies among players, leading to a more accurate model by accounting for interaction-like effects. Complexity: Increases the number of parameters to estimate, including a variance component for the random slopes.
model3 <- lmer(Reaction_Time ~ 1 + Phrase_Condition + (1 + Phrase_Condition | Player_ID), data = data)## Warning in checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv, :
## Model failed to converge with max|grad| = 0.00768178 (tol = 0.002, component 1)# Obtain and print a summary of the model to examine fixed and random effects
print(summary(model3))## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: Reaction_Time ~ 1 + Phrase_Condition + (1 + Phrase_Condition |
## Player_ID)
## Data: data
##
## REML criterion at convergence: 62721.8
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -2.566 -0.342 -0.103 0.174 48.516
##
## Random effects:
## Groups Name Variance Std.Dev. Corr
## Player_ID (Intercept) 0.008713 0.09334
## Phrase_Condition2 0.031595 0.17775 0.29
## Phrase_Condition3 0.026379 0.16242 0.07 0.62
## Phrase_Condition4 0.098637 0.31407 0.40 0.20 -0.04
## Residual 0.950765 0.97507
## Number of obs: 22404, groups: Player_ID, 38
##
## Fixed effects:
## Estimate Std. Error df t value Pr(>|t|)
## (Intercept) 0.67601 0.02165 38.30934 31.227 < 2e-16 ***
## Phrase_Condition2 0.26567 0.03629 36.93804 7.322 1.07e-08 ***
## Phrase_Condition3 0.05920 0.03407 37.95185 1.738 0.0904 .
## Phrase_Condition4 0.51368 0.05429 38.00528 9.462 1.55e-11 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) Phr_C2 Phr_C3
## Phrs_Cndtn2 -0.143
## Phrs_Cndtn3 -0.288 0.577
## Phrs_Cndtn4 0.059 0.267 0.101
## optimizer (nloptwrap) convergence code: 0 (OK)
## Model failed to converge with max|grad| = 0.00768178 (tol = 0.002, component 1)#Diagnostic plots
plot_residuals_diagnostics(model3)
## `geom_smooth()` using formula = 'y ~ x'
# Diagnostic checks - Simulating residuals to validate model assumptions
residuals_simulation <- simulateResiduals(fittedModel = model3, n = 500)
plot(residuals_simulation)
# Model diagnostics
check_model(model3)
# Check for influential cases potentially affecting the model
#influence_measures <- influence(model3)
#plot(influence_measures, which = "cook")
# If assumptions are not met, consider re-fitting the model with transformations or different specifications
# model_revised <- update(model, . ~ . + log(Reaction_Time))
# print(summary(model_revised))
# Model interpretation - examining fixed and random effects
#fixed_effects <- fixef(model3)
#random_effects <- ranef(model3)
# Printing fixed effects for interpretation
#print(fixed_effects)
# Printing random effects for interpretation
#print(random_effects)
# Conducting pairwise comparisons of phrase conditions using estimated marginal means
emm_results <- emmeans(model3, specs = ~ Phrase_Condition)## Note: D.f. calculations have been disabled because the number of observations exceeds 3000.
## To enable adjustments, add the argument 'pbkrtest.limit = 22404' (or larger)
## [or, globally, 'set emm_options(pbkrtest.limit = 22404)' or larger];
## but be warned that this may result in large computation time and memory use.
## Note: D.f. calculations have been disabled because the number of observations exceeds 3000.
## To enable adjustments, add the argument 'lmerTest.limit = 22404' (or larger)
## [or, globally, 'set emm_options(lmerTest.limit = 22404)' or larger];
## but be warned that this may result in large computation time and memory use.summary(emm_results)## Phrase_Condition emmean SE df asymp.LCL asymp.UCL
## 1 0.676 0.0216 Inf 0.634 0.718
## 2 0.942 0.0395 Inf 0.864 1.019
## 3 0.735 0.0347 Inf 0.667 0.803
## 4 1.190 0.0596 Inf 1.073 1.307
##
## Degrees-of-freedom method: asymptotic
## Confidence level used: 0.95pairwise_comparisons <- emmeans(model3, specs = pairwise ~ Phrase_Condition)## Note: D.f. calculations have been disabled because the number of observations exceeds 3000.
## To enable adjustments, add the argument 'pbkrtest.limit = 22404' (or larger)
## [or, globally, 'set emm_options(pbkrtest.limit = 22404)' or larger];
## but be warned that this may result in large computation time and memory use.
## Note: D.f. calculations have been disabled because the number of observations exceeds 3000.
## To enable adjustments, add the argument 'lmerTest.limit = 22404' (or larger)
## [or, globally, 'set emm_options(lmerTest.limit = 22404)' or larger];
## but be warned that this may result in large computation time and memory use.summary(pairwise_comparisons)## $emmeans
## Phrase_Condition emmean SE df asymp.LCL asymp.UCL
## 1 0.676 0.0216 Inf 0.634 0.718
## 2 0.942 0.0395 Inf 0.864 1.019
## 3 0.735 0.0347 Inf 0.667 0.803
## 4 1.190 0.0596 Inf 1.073 1.307
##
## Degrees-of-freedom method: asymptotic
## Confidence level used: 0.95
##
## $contrasts
## contrast estimate SE df z.ratio p.value
## Phrase_Condition1 - Phrase_Condition2 -0.2657 0.0363 Inf -7.322 <.0001
## Phrase_Condition1 - Phrase_Condition3 -0.0592 0.0341 Inf -1.738 0.3041
## Phrase_Condition1 - Phrase_Condition4 -0.5137 0.0543 Inf -9.462 <.0001
## Phrase_Condition2 - Phrase_Condition3 0.2065 0.0324 Inf 6.367 <.0001
## Phrase_Condition2 - Phrase_Condition4 -0.2480 0.0567 Inf -4.376 0.0001
## Phrase_Condition3 - Phrase_Condition4 -0.4545 0.0611 Inf -7.439 <.0001
##
## Degrees-of-freedom method: asymptotic
## P value adjustment: tukey method for comparing a family of 4 estimatesplot(emm_results)
Overall Model Interpretation: This model shows how different phrase
conditions affect reaction times, accounting for both average player
effects and individual differences. Conditions 2 and 4 significantly
increase reaction times, suggesting they are more challenging. This can
guide game difficulty adjustments or targeted practice for players
struggling with specific phrases. The random slopes indicate that these
conditions impact players differently, supporting personalized gameplay
or learning approaches.
Condition Effects:
Phrase_Condition2: Increases reaction time by 0.23230 seconds (p < 1.27e-10). Phrase_Condition3: Increases reaction time by 0.03446 seconds, not statistically significant (p = 0.106). Phrase_Condition4: Increases reaction time by 0.39736 seconds (p < 1.74e-14). Phrase_Condition4 is the hardest, with the largest significant effect, followed by Phrase_Condition2. Phrase_Condition3 does not significantly differ from the baseline.
# Create a boxplot
ggplot(data, aes(x = Phrase_Condition, y = Reaction_Time, fill = Phrase_Condition)) +
geom_boxplot() +
labs(title = "Reaction Times by Phrase Condition", x = "Phrase Condition", y = "Reaction Time (seconds)") +
theme_minimal()
# Calculate average reaction times per player per condition
avg_reaction_times <- data %>%
group_by(Player_ID, Phrase_Condition) %>%
summarise(Avg_Reaction_Time = mean(Reaction_Time), .groups = 'drop')
# Interaction plot
ggplot(avg_reaction_times, aes(x = Phrase_Condition, y = Avg_Reaction_Time, group = Player_ID, color = Phrase_Condition)) +
geom_line(alpha = 0.5) + # Slightly transparent to see overlapping lines
geom_point() +
labs(title = "Interaction of Phrase Condition and Player on Reaction Times", x = "Phrase Condition", y = "Average Reaction Time (seconds)") +
theme_minimal()
Jorge: ## 3. Does the version of the experiment cause different outcomes?
model8 <- lmer(Reaction_Time ~ 1 + Game_Version + (1 | Player_ID), data = data)
# Obtain and print a summary of the model to examine fixed and random effects
print(summary(model8))## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: Reaction_Time ~ 1 + Game_Version + (1 | Player_ID)
## Data: data
##
## REML criterion at convergence: 64210.2
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -1.468 -0.389 -0.111 0.199 46.837
##
## Random effects:
## Groups Name Variance Std.Dev.
## Player_ID (Intercept) 0.04381 0.2093
## Residual 1.02245 1.0112
## Number of obs: 22404, groups: Player_ID, 38
##
## Fixed effects:
## Estimate Std. Error df t value Pr(>|t|)
## (Intercept) 0.85459 0.07143 34.55240 11.964 7.7e-14 ***
## Game_Version2 0.17590 0.10093 34.44373 1.743 0.0903 .
## Game_Version3 0.15994 0.09826 34.27187 1.628 0.1127
## Game_Version4 0.06567 0.09822 34.22056 0.669 0.5082
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) Gm_Vr2 Gm_Vr3
## Game_Versn2 -0.708
## Game_Versn3 -0.727 0.514
## Game_Versn4 -0.727 0.515 0.529#Diagnostic plots
plot_residuals_diagnostics(model8)
## `geom_smooth()` using formula = 'y ~ x'
# Diagnostic checks - Simulating residuals to validate model assumptions
residuals_simulation <- simulateResiduals(fittedModel = model8, n = 500)
plot(residuals_simulation)
# Model diagnostics
check_model(model8)
# Check for influential cases potentially affecting the model
#influence_measures <- influence(model4)
#plot(influence_measures, which = "cook")
# If assumptions are not met, consider re-fitting the model with transformations or different specifications
# model_revised <- update(model, . ~ . + log(Reaction_Time))
# print(summary(model_revised))
# Model interpretation - examining fixed and random effects
fixed_effects <- fixef(model8)
random_effects <- ranef(model8)How does the effect of condition vary depending on the version?
4b: rt ~ 1 + condition * version + (1 + condition | participant)
Main Effects:
Phrase_Condition: Measures how different phrase conditions affect reaction times compared to the baseline condition. Game_Version: Evaluates the impact of different game versions on reaction times relative to the baseline version. Interaction Effect (Phrase_Condition * Game_Version):
Assesses if the effect of a phrase condition on reaction times changes across game versions. Each interaction coefficient shows how the combined effect of a phrase condition and game version differs from their individual effects. Random Effects:
(1 | Player_ID): Models random intercepts for each player, indicating variability in baseline reaction times not explained by phrase conditions or game versions. Each player has an adjusted baseline reaction time. Interpretation:
Intercept: Represents the average reaction time for the baseline phrase condition and game version. Main Effects: Coefficients show how reaction times change with each phrase condition and game version, assuming no interaction. Interaction Effects: Significant interaction terms indicate that the effect of a phrase condition on reaction time is modified by the game version, suggesting areas for tailored game design to optimize learning or user experience.
#This doesnt work:
#model4 <- lmer(Reaction_Time ~ 1 + Phrase_Condition * Game_Version + (1 + Phrase_Condition | Player_ID), data = data)
model4 <- lmer(Reaction_Time ~ 1 + Phrase_Condition * Game_Version + (1 | Player_ID), data = data)
# Obtain and print a summary of the model to examine fixed and random effects
print(summary(model4))## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: Reaction_Time ~ 1 + Phrase_Condition * Game_Version + (1 | Player_ID)
## Data: data
##
## REML criterion at convergence: 63270.8
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -1.755 -0.357 -0.110 0.181 47.728
##
## Random effects:
## Groups Name Variance Std.Dev.
## Player_ID (Intercept) 0.04723 0.2173
## Residual 0.97828 0.9891
## Number of obs: 22404, groups: Player_ID, 38
##
## Fixed effects:
## Estimate Std. Error df t value
## (Intercept) 7.178e-01 7.988e-02 4.692e+01 8.987
## Phrase_Condition2 1.388e-01 4.755e-02 2.235e+04 2.919
## Phrase_Condition3 -8.641e-02 4.672e-02 2.236e+04 -1.849
## Phrase_Condition4 3.243e-01 4.122e-02 2.237e+04 7.867
## Game_Version2 -3.493e-02 1.124e-01 4.596e+01 -0.311
## Game_Version3 -8.644e-02 1.100e-01 4.673e+01 -0.786
## Game_Version4 -3.894e-02 1.093e-01 4.553e+01 -0.356
## Phrase_Condition2:Game_Version2 3.150e-01 6.539e-02 2.236e+04 4.817
## Phrase_Condition3:Game_Version2 2.283e-01 6.441e-02 2.236e+04 3.545
## Phrase_Condition4:Game_Version2 2.647e-01 5.685e-02 2.237e+04 4.657
## Phrase_Condition2:Game_Version3 2.159e-01 6.514e-02 2.235e+04 3.314
## Phrase_Condition3:Game_Version3 3.713e-01 6.309e-02 2.236e+04 5.885
## Phrase_Condition4:Game_Version3 2.459e-01 5.515e-02 2.237e+04 4.459
## Phrase_Condition2:Game_Version4 -6.528e-03 6.312e-02 2.236e+04 -0.103
## Phrase_Condition3:Game_Version4 7.052e-02 6.188e-02 2.236e+04 1.140
## Phrase_Condition4:Game_Version4 1.493e-01 5.333e-02 2.237e+04 2.799
## Pr(>|t|)
## (Intercept) 9.09e-12 ***
## Phrase_Condition2 0.003512 **
## Phrase_Condition3 0.064404 .
## Phrase_Condition4 3.80e-15 ***
## Game_Version2 0.757347
## Game_Version3 0.435910
## Game_Version4 0.723223
## Phrase_Condition2:Game_Version2 1.47e-06 ***
## Phrase_Condition3:Game_Version2 0.000394 ***
## Phrase_Condition4:Game_Version2 3.23e-06 ***
## Phrase_Condition2:Game_Version3 0.000921 ***
## Phrase_Condition3:Game_Version3 4.03e-09 ***
## Phrase_Condition4:Game_Version3 8.28e-06 ***
## Phrase_Condition2:Game_Version4 0.917623
## Phrase_Condition3:Game_Version4 0.254420
## Phrase_Condition4:Game_Version4 0.005133 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1##
## Correlation matrix not shown by default, as p = 16 > 12.
## Use print(summary(model4), correlation=TRUE) or
## vcov(summary(model4)) if you need it#Diagnostic plots
plot_residuals_diagnostics(model4)
## `geom_smooth()` using formula = 'y ~ x'
# Diagnostic checks - Simulating residuals to validate model assumptions
residuals_simulation <- simulateResiduals(fittedModel = model4, n = 500)
plot(residuals_simulation)
# Model diagnostics
check_model(model4)
# Check for influential cases potentially affecting the model
#influence_measures <- influence(model4)
#plot(influence_measures, which = "cook")
# If assumptions are not met, consider re-fitting the model with transformations or different specifications
# model_revised <- update(model, . ~ . + log(Reaction_Time))
# print(summary(model_revised))
# Model interpretation - examining fixed and random effects
fixed_effects <- fixef(model4)
random_effects <- ranef(model4)
# Printing fixed effects for interpretation
print(fixed_effects)## (Intercept) Phrase_Condition2
## 0.717835648 0.138799455
## Phrase_Condition3 Phrase_Condition4
## -0.086411190 0.324269247
## Game_Version2 Game_Version3
## -0.034929318 -0.086438000
## Game_Version4 Phrase_Condition2:Game_Version2
## -0.038941112 0.314959990
## Phrase_Condition3:Game_Version2 Phrase_Condition4:Game_Version2
## 0.228307728 0.264739313
## Phrase_Condition2:Game_Version3 Phrase_Condition3:Game_Version3
## 0.215875147 0.371271465
## Phrase_Condition4:Game_Version3 Phrase_Condition2:Game_Version4
## 0.245890992 -0.006528468
## Phrase_Condition3:Game_Version4 Phrase_Condition4:Game_Version4
## 0.070520852 0.149258028# Printing random effects for interpretation
print(random_effects)## $Player_ID
## (Intercept)
## 1 -0.193400990
## 2 -0.172907500
## 3 -0.139512409
## 4 0.054738806
## 5 0.097357400
## 6 0.007593549
## 7 0.169121999
## 8 -0.015378239
## 9 0.192387383
## 10 -0.040921626
## 11 -0.119072729
## 12 0.277477969
## 13 -0.138723069
## 14 -0.088199447
## 15 0.298225096
## 16 0.149117761
## 17 -0.379697071
## 18 0.041793116
## 19 0.091510015
## 20 -0.109400826
## 21 -0.038363901
## 22 0.014469477
## 23 -0.279909985
## 24 0.106276801
## 25 0.341888139
## 26 0.134232377
## 27 -0.152810008
## 28 -0.107892089
## 29 0.152628995
## 30 -0.098880689
## 31 0.083349923
## 32 0.021809041
## 33 0.653097901
## 34 -0.288705849
## 35 0.136633493
## 36 -0.186142609
## 37 -0.300622841
## 38 -0.173167365
##
## with conditional variances for "Player_ID"# Estimate marginal means for all combinations
emm1 <- emmeans(model4, specs = ~ Phrase_Condition * Game_Version)## Note: D.f. calculations have been disabled because the number of observations exceeds 3000.
## To enable adjustments, add the argument 'pbkrtest.limit = 22404' (or larger)
## [or, globally, 'set emm_options(pbkrtest.limit = 22404)' or larger];
## but be warned that this may result in large computation time and memory use.## Note: D.f. calculations have been disabled because the number of observations exceeds 3000.
## To enable adjustments, add the argument 'lmerTest.limit = 22404' (or larger)
## [or, globally, 'set emm_options(lmerTest.limit = 22404)' or larger];
## but be warned that this may result in large computation time and memory use.summary(emm1)## Phrase_Condition Game_Version emmean SE df asymp.LCL asymp.UCL
## 1 1 0.718 0.0799 Inf 0.561 0.874
## 2 1 0.857 0.0799 Inf 0.700 1.013
## 3 1 0.631 0.0794 Inf 0.476 0.787
## 4 1 1.042 0.0763 Inf 0.893 1.192
## 1 2 0.683 0.0791 Inf 0.528 0.838
## 2 2 1.137 0.0792 Inf 0.981 1.292
## 3 2 0.825 0.0789 Inf 0.670 0.979
## 4 2 1.272 0.0761 Inf 1.123 1.421
## 1 3 0.631 0.0756 Inf 0.483 0.780
## 2 3 0.986 0.0756 Inf 0.838 1.134
## 3 3 0.916 0.0743 Inf 0.771 1.062
## 4 3 1.202 0.0712 Inf 1.062 1.341
## 1 4 0.679 0.0746 Inf 0.533 0.825
## 2 4 0.811 0.0749 Inf 0.664 0.958
## 3 4 0.663 0.0744 Inf 0.517 0.809
## 4 4 1.152 0.0710 Inf 1.013 1.292
##
## Degrees-of-freedom method: asymptotic
## Confidence level used: 0.95# Pairwise comparisons across phrase conditions within each game version
pairs(emm1, simple = "each")## $`simple contrasts for Phrase_Condition`
## Game_Version = 1:
## contrast estimate SE df z.ratio p.value
## Phrase_Condition1 - Phrase_Condition2 -0.1388 0.0475 Inf -2.919 0.0184
## Phrase_Condition1 - Phrase_Condition3 0.0864 0.0467 Inf 1.849 0.2502
## Phrase_Condition1 - Phrase_Condition4 -0.3243 0.0412 Inf -7.867 <.0001
## Phrase_Condition2 - Phrase_Condition3 0.2252 0.0467 Inf 4.824 <.0001
## Phrase_Condition2 - Phrase_Condition4 -0.1855 0.0412 Inf -4.504 <.0001
## Phrase_Condition3 - Phrase_Condition4 -0.4107 0.0400 Inf -10.260 <.0001
##
## Game_Version = 2:
## contrast estimate SE df z.ratio p.value
## Phrase_Condition1 - Phrase_Condition2 -0.4538 0.0449 Inf -10.108 <.0001
## Phrase_Condition1 - Phrase_Condition3 -0.1419 0.0443 Inf -3.201 0.0075
## Phrase_Condition1 - Phrase_Condition4 -0.5890 0.0392 Inf -15.044 <.0001
## Phrase_Condition2 - Phrase_Condition3 0.3119 0.0446 Inf 6.992 <.0001
## Phrase_Condition2 - Phrase_Condition4 -0.1352 0.0396 Inf -3.412 0.0036
## Phrase_Condition3 - Phrase_Condition4 -0.4471 0.0389 Inf -11.494 <.0001
##
## Game_Version = 3:
## contrast estimate SE df z.ratio p.value
## Phrase_Condition1 - Phrase_Condition2 -0.3547 0.0445 Inf -7.966 <.0001
## Phrase_Condition1 - Phrase_Condition3 -0.2849 0.0424 Inf -6.720 <.0001
## Phrase_Condition1 - Phrase_Condition4 -0.5702 0.0366 Inf -15.562 <.0001
## Phrase_Condition2 - Phrase_Condition3 0.0698 0.0423 Inf 1.650 0.3505
## Phrase_Condition2 - Phrase_Condition4 -0.2155 0.0365 Inf -5.897 <.0001
## Phrase_Condition3 - Phrase_Condition4 -0.2853 0.0337 Inf -8.460 <.0001
##
## Game_Version = 4:
## contrast estimate SE df z.ratio p.value
## Phrase_Condition1 - Phrase_Condition2 -0.1323 0.0415 Inf -3.186 0.0079
## Phrase_Condition1 - Phrase_Condition3 0.0159 0.0406 Inf 0.392 0.9796
## Phrase_Condition1 - Phrase_Condition4 -0.4735 0.0338 Inf -13.994 <.0001
## Phrase_Condition2 - Phrase_Condition3 0.1482 0.0412 Inf 3.598 0.0018
## Phrase_Condition2 - Phrase_Condition4 -0.3413 0.0347 Inf -9.845 <.0001
## Phrase_Condition3 - Phrase_Condition4 -0.4894 0.0330 Inf -14.833 <.0001
##
## Degrees-of-freedom method: asymptotic
## P value adjustment: tukey method for comparing a family of 4 estimates
##
## $`simple contrasts for Game_Version`
## Phrase_Condition = 1:
## contrast estimate SE df z.ratio p.value
## Game_Version1 - Game_Version2 0.03493 0.112 Inf 0.311 0.9896
## Game_Version1 - Game_Version3 0.08644 0.110 Inf 0.786 0.8609
## Game_Version1 - Game_Version4 0.03894 0.109 Inf 0.356 0.9845
## Game_Version2 - Game_Version3 0.05151 0.109 Inf 0.471 0.9655
## Game_Version2 - Game_Version4 0.00401 0.109 Inf 0.037 1.0000
## Game_Version3 - Game_Version4 -0.04750 0.106 Inf -0.447 0.9702
##
## Phrase_Condition = 2:
## contrast estimate SE df z.ratio p.value
## Game_Version1 - Game_Version2 -0.28003 0.112 Inf -2.489 0.0615
## Game_Version1 - Game_Version3 -0.12944 0.110 Inf -1.177 0.6412
## Game_Version1 - Game_Version4 0.04547 0.109 Inf 0.415 0.9759
## Game_Version2 - Game_Version3 0.15059 0.109 Inf 1.375 0.5149
## Game_Version2 - Game_Version4 0.32550 0.109 Inf 2.985 0.0150
## Game_Version3 - Game_Version4 0.17491 0.106 Inf 1.644 0.3540
##
## Phrase_Condition = 3:
## contrast estimate SE df z.ratio p.value
## Game_Version1 - Game_Version2 -0.19338 0.112 Inf -1.728 0.3090
## Game_Version1 - Game_Version3 -0.28483 0.109 Inf -2.619 0.0437
## Game_Version1 - Game_Version4 -0.03158 0.109 Inf -0.290 0.9915
## Game_Version2 - Game_Version3 -0.09146 0.108 Inf -0.844 0.8335
## Game_Version2 - Game_Version4 0.16180 0.108 Inf 1.492 0.4424
## Game_Version3 - Game_Version4 0.25325 0.105 Inf 2.408 0.0756
##
## Phrase_Condition = 4:
## contrast estimate SE df z.ratio p.value
## Game_Version1 - Game_Version2 -0.22981 0.108 Inf -2.134 0.1425
## Game_Version1 - Game_Version3 -0.15945 0.104 Inf -1.528 0.4204
## Game_Version1 - Game_Version4 -0.11032 0.104 Inf -1.059 0.7148
## Game_Version2 - Game_Version3 0.07036 0.104 Inf 0.675 0.9065
## Game_Version2 - Game_Version4 0.11949 0.104 Inf 1.148 0.6597
## Game_Version3 - Game_Version4 0.04914 0.101 Inf 0.489 0.9617
##
## Degrees-of-freedom method: asymptotic
## P value adjustment: tukey method for comparing a family of 4 estimatesplot(emm1)
- Phrase_Condition4 shows a substantial
increase in reaction times, suggesting it is more challenging than the
baseline condition. - Interactions such as
Phrase_Condition2:Game_Version2 and
Phrase_Condition3:Game_Version3 are significantly
affecting reaction times, indicating that the combination of these
conditions and versions exacerbates or alters the impact seen with the
conditions alone. - Non-significant main effects for
Game_Version (2, 3, and 4) suggest that changes in
game versions alone do not significantly alter reaction times, but their
interactions with phrase conditions do. - Practical
Significance: The model indicates that while some game versions
and phrase conditions significantly affect reaction times, others do
not. This can inform adjustments in game design or targeted
interventions for training or improvement purposes. - Model Fit
and Quality: The REML criterion and scaled residuals indicate
the model fits the data well. Scaled residuals spread mostly
symmetrically around zero, suggesting adequate model fit without
apparent bias.
5. Does the rate of improvement in each condition depend on the version?
5a: rt ~ 1 + time + condition * version + (1 + time + condition | participant)
1 (Intercept): Represents the baseline reaction time for the first question and the baseline categories of Phrase_Condition and Game_Version.
Question_Num: Continuous variable representing the sequential number of questions. A negative coefficient indicates improved reaction time (i.e., decreasing reaction time) as the game progresses.
Phrase_Condition * Game_Version: Examines if the effect of phrase conditions on reaction time is modified by the game version, identifying combinations where game version influences the difficulty of phrase conditions.
(1 | Player_ID): Random intercepts for players account for individual differences in baseline reaction times, acknowledging that some players generally respond faster or slower than others, independent of condition or version.
model5 <- lmer(Reaction_Time ~ 1 + Question_Num + Phrase_Condition * Game_Version + (1 | Player_ID), data = data)
# Obtain and print a summary of the model to examine fixed and random effects
print(summary(model5))## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: Reaction_Time ~ 1 + Question_Num + Phrase_Condition * Game_Version +
## (1 | Player_ID)
## Data: data
##
## REML criterion at convergence: 63274.3
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -1.766 -0.358 -0.108 0.184 47.738
##
## Random effects:
## Groups Name Variance Std.Dev.
## Player_ID (Intercept) 0.0474 0.2177
## Residual 0.9777 0.9888
## Number of obs: 22404, groups: Player_ID, 38
##
## Fixed effects:
## Estimate Std. Error df t value
## (Intercept) 7.535e-01 8.058e-02 4.824e+01 9.352
## Question_Num -2.867e-04 7.781e-05 2.236e+04 -3.685
## Phrase_Condition2 1.457e-01 4.757e-02 2.235e+04 3.064
## Phrase_Condition3 -5.557e-02 4.745e-02 2.236e+04 -1.171
## Phrase_Condition4 3.565e-01 4.213e-02 2.237e+04 8.463
## Game_Version2 3.184e-03 1.130e-01 4.668e+01 0.028
## Game_Version3 -5.439e-02 1.105e-01 4.726e+01 -0.492
## Game_Version4 -3.190e-02 1.095e-01 4.551e+01 -0.291
## Phrase_Condition2:Game_Version2 2.708e-01 6.646e-02 2.235e+04 4.075
## Phrase_Condition3:Game_Version2 1.652e-01 6.662e-02 2.236e+04 2.480
## Phrase_Condition4:Game_Version2 2.166e-01 5.832e-02 2.237e+04 3.714
## Phrase_Condition2:Game_Version3 2.151e-01 6.512e-02 2.235e+04 3.304
## Phrase_Condition3:Game_Version3 3.068e-01 6.545e-02 2.236e+04 4.688
## Phrase_Condition4:Game_Version3 1.751e-01 5.838e-02 2.237e+04 2.999
## Phrase_Condition2:Game_Version4 1.060e-02 6.327e-02 2.236e+04 0.168
## Phrase_Condition3:Game_Version4 7.043e-02 6.186e-02 2.236e+04 1.139
## Phrase_Condition4:Game_Version4 9.266e-02 5.548e-02 2.237e+04 1.670
## Pr(>|t|)
## (Intercept) 2.06e-12 ***
## Question_Num 0.000229 ***
## Phrase_Condition2 0.002189 **
## Phrase_Condition3 0.241561
## Phrase_Condition4 < 2e-16 ***
## Game_Version2 0.977645
## Game_Version3 0.624813
## Game_Version4 0.772057
## Phrase_Condition2:Game_Version2 4.62e-05 ***
## Phrase_Condition3:Game_Version2 0.013153 *
## Phrase_Condition4:Game_Version2 0.000204 ***
## Phrase_Condition2:Game_Version3 0.000955 ***
## Phrase_Condition3:Game_Version3 2.77e-06 ***
## Phrase_Condition4:Game_Version3 0.002709 **
## Phrase_Condition2:Game_Version4 0.866902
## Phrase_Condition3:Game_Version4 0.254880
## Phrase_Condition4:Game_Version4 0.094913 .
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1##
## Correlation matrix not shown by default, as p = 17 > 12.
## Use print(summary(model5), correlation=TRUE) or
## vcov(summary(model5)) if you need it#Diagnostic plots
plot_residuals_diagnostics(model5)
## `geom_smooth()` using formula = 'y ~ x'
# Diagnostic checks - Simulating residuals to validate model assumptions
residuals_simulation <- simulateResiduals(fittedModel = model5, n = 500)
plot(residuals_simulation)
# Model diagnostics
check_model(model5)
# Check for influential cases potentially affecting the model
#influence_measures <- influence(model5)
#plot(influence_measures, which = "cook")
# If assumptions are not met, consider re-fitting the model with transformations or different specifications
# model_revised <- update(model, . ~ . + log(Reaction_Time))
# print(summary(model_revised))
# Model interpretation - examining fixed and random effects
fixed_effects <- fixef(model5)
random_effects <- ranef(model5)
# Printing fixed effects for interpretation
print(fixed_effects)## (Intercept) Question_Num
## 0.7535226350 -0.0002867384
## Phrase_Condition2 Phrase_Condition3
## 0.1457410256 -0.0555730125
## Phrase_Condition4 Game_Version2
## 0.3565391405 0.0031839946
## Game_Version3 Game_Version4
## -0.0543915089 -0.0318970820
## Phrase_Condition2:Game_Version2 Phrase_Condition3:Game_Version2
## 0.2708267333 0.1652151615
## Phrase_Condition4:Game_Version2 Phrase_Condition2:Game_Version3
## 0.2166081933 0.2151443866
## Phrase_Condition3:Game_Version3 Phrase_Condition4:Game_Version3
## 0.3068451377 0.1751056415
## Phrase_Condition2:Game_Version4 Phrase_Condition3:Game_Version4
## 0.0106042643 0.0704325692
## Phrase_Condition4:Game_Version4
## 0.0926615960# Printing random effects for interpretation
print(random_effects)## $Player_ID
## (Intercept)
## 1 -0.192754145
## 2 -0.174395194
## 3 -0.138741453
## 4 0.055547188
## 5 0.097623412
## 6 0.006281094
## 7 0.169758781
## 8 -0.015250655
## 9 0.191930974
## 10 -0.039913031
## 11 -0.118079192
## 12 0.277690794
## 13 -0.137919769
## 14 -0.088200433
## 15 0.298405759
## 16 0.146284999
## 17 -0.382323726
## 18 0.044054599
## 19 0.087427025
## 20 -0.110594339
## 21 -0.039061676
## 22 0.018831239
## 23 -0.276981734
## 24 0.103730625
## 25 0.340950792
## 26 0.134942965
## 27 -0.151602715
## 28 -0.107642183
## 29 0.155285600
## 30 -0.099797419
## 31 0.082875172
## 32 0.020498780
## 33 0.653603944
## 34 -0.290253687
## 35 0.143599566
## 36 -0.187536411
## 37 -0.306304699
## 38 -0.171970848
##
## with conditional variances for "Player_ID"# Estimate marginal means at various levels of Question_Num
emm2 <- emmeans(model5, specs = ~ Phrase_Condition * Game_Version, at = list(Question_Num = c(1, 10, 20)))## Note: D.f. calculations have been disabled because the number of observations exceeds 3000.
## To enable adjustments, add the argument 'pbkrtest.limit = 22404' (or larger)
## [or, globally, 'set emm_options(pbkrtest.limit = 22404)' or larger];
## but be warned that this may result in large computation time and memory use.## Note: D.f. calculations have been disabled because the number of observations exceeds 3000.
## To enable adjustments, add the argument 'lmerTest.limit = 22404' (or larger)
## [or, globally, 'set emm_options(lmerTest.limit = 22404)' or larger];
## but be warned that this may result in large computation time and memory use.summary(emm2)## Phrase_Condition Game_Version emmean SE df asymp.LCL asymp.UCL
## 1 1 0.751 0.0805 Inf 0.593 0.908
## 2 1 0.896 0.0807 Inf 0.738 1.054
## 3 1 0.695 0.0813 Inf 0.536 0.854
## 4 1 1.107 0.0784 Inf 0.953 1.261
## 1 2 0.754 0.0815 Inf 0.594 0.913
## 2 2 1.170 0.0799 Inf 1.014 1.327
## 3 2 0.863 0.0797 Inf 0.707 1.020
## 4 2 1.327 0.0776 Inf 1.175 1.479
## 1 3 0.696 0.0777 Inf 0.544 0.849
## 2 3 1.057 0.0781 Inf 0.904 1.210
## 3 3 0.947 0.0749 Inf 0.801 1.094
## 4 3 1.228 0.0717 Inf 1.087 1.368
## 1 4 0.719 0.0755 Inf 0.571 0.867
## 2 4 0.875 0.0770 Inf 0.724 1.026
## 3 4 0.734 0.0770 Inf 0.583 0.884
## 4 4 1.168 0.0713 Inf 1.028 1.308
##
## Results are averaged over the levels of: Question_Num
## Degrees-of-freedom method: asymptotic
## Confidence level used: 0.95# Trend analysis
trend_analysis <- emtrends(model5, specs = ~ Phrase_Condition * Game_Version, var = "Question_Num")## Note: D.f. calculations have been disabled because the number of observations exceeds 3000.
## To enable adjustments, add the argument 'pbkrtest.limit = 22404' (or larger)
## [or, globally, 'set emm_options(pbkrtest.limit = 22404)' or larger];
## but be warned that this may result in large computation time and memory use.
## Note: D.f. calculations have been disabled because the number of observations exceeds 3000.
## To enable adjustments, add the argument 'lmerTest.limit = 22404' (or larger)
## [or, globally, 'set emm_options(lmerTest.limit = 22404)' or larger];
## but be warned that this may result in large computation time and memory use.summary(trend_analysis)## Phrase_Condition Game_Version Question_Num.trend SE df asymp.LCL
## 1 1 -0.000287 7.78e-05 Inf -0.000439
## 2 1 -0.000287 7.78e-05 Inf -0.000439
## 3 1 -0.000287 7.78e-05 Inf -0.000439
## 4 1 -0.000287 7.78e-05 Inf -0.000439
## 1 2 -0.000287 7.78e-05 Inf -0.000439
## 2 2 -0.000287 7.78e-05 Inf -0.000439
## 3 2 -0.000287 7.78e-05 Inf -0.000439
## 4 2 -0.000287 7.78e-05 Inf -0.000439
## 1 3 -0.000287 7.78e-05 Inf -0.000439
## 2 3 -0.000287 7.78e-05 Inf -0.000439
## 3 3 -0.000287 7.78e-05 Inf -0.000439
## 4 3 -0.000287 7.78e-05 Inf -0.000439
## 1 4 -0.000287 7.78e-05 Inf -0.000439
## 2 4 -0.000287 7.78e-05 Inf -0.000439
## 3 4 -0.000287 7.78e-05 Inf -0.000439
## 4 4 -0.000287 7.78e-05 Inf -0.000439
## asymp.UCL
## -0.000134
## -0.000134
## -0.000134
## -0.000134
## -0.000134
## -0.000134
## -0.000134
## -0.000134
## -0.000134
## -0.000134
## -0.000134
## -0.000134
## -0.000134
## -0.000134
## -0.000134
## -0.000134
##
## Degrees-of-freedom method: asymptotic
## Confidence level used: 0.95plot(emm2)
Model Overview
Formula: Reaction time ~ question number, phrase conditions, game versions, and their interactions, with a random intercept for each player. Data: 21,488 observations from 38 unique players. Random Effects
Player Variability: Variance = 0.01431 (Std.Dev. = 0.1196), indicating moderate variation. Residual Variability: Variance = 0.13911 (Std.Dev. = 0.3730), suggesting much of the variation remains unexplained. Fixed Effects
Intercept: Baseline reaction time is 0.6381 seconds. Question_Num: Positive but very small coefficient (1.384e-04), indicating minimal change over time. Phrase_Condition: Phrase_Condition4 significantly increases reaction times; Phrase_Condition2 increases slightly; Phrase_Condition3 shows a minor decrease. Game_Version: Versions 2, 3, and 4 do not significantly affect reaction times. Interaction Effects
Significant Positive Interactions: Certain combinations (e.g., Phrase_Condition3 and Game_Version3, Phrase_Condition2 and Game_Version2) increase reaction times more than individual effects. Non-significant/Negative Interactions: Some combinations do not significantly impact or slightly decrease reaction times (e.g., Phrase_Condition4). Interpretation and Implications
Interactions Matter: Highlights the importance of considering interactions between game design elements. Minimal Overall Time Trend: Very small positive coefficient for Question_Num suggests minimal change in reaction times, indicating little learning or fatigue effects. Model Fit and Residuals: Significant random effects variance indicates the model captures some complexity, but individual differences and unmodeled factors still contribute to variability in reaction times.
Let’s compare different models
# Fit the models as described
model6 <- lmer(Reaction_Time ~ 1 + Phrase_Condition * Game_Version + (1 | Player_ID), data = data)
# Model with random intercept and random slope for the interaction term
model7 <- lmer(Reaction_Time ~ 1 + Phrase_Condition * Game_Version + (1 + Phrase_Condition | Player_ID), data = data)## boundary (singular) fit: see help('isSingular')## Warning: Model failed to converge with 1 negative eigenvalue: -5.2e+01# Model with random intercept and random slope for the interaction term
#model8 <- lmer(Reaction_Time ~ 1 + Phrase_Condition * Game_Version +
# (1 | Player_ID) + (0 + Phrase_Condition * Game_Version | Player_ID), data = data)
anova(model6, model7)## refitting model(s) with ML (instead of REML)## Data: data
## Models:
## model6: Reaction_Time ~ 1 + Phrase_Condition * Game_Version + (1 | Player_ID)
## model7: Reaction_Time ~ 1 + Phrase_Condition * Game_Version + (1 + Phrase_Condition | Player_ID)
## npar AIC BIC logLik deviance Chisq Df Pr(>Chisq)
## model6 18 63235 63379 -31599 63199
## model7 27 62720 62936 -31333 62666 533.22 9 < 2.2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Interpretation 6: This model tells you how Phrase_Condition and Game_Version interact to influence reaction time, considering the average differences among players.
Interpretation 7: This model tells you how Phrase_Condition and Game_Version interact to influence reaction time, and also how the effect of Phrase_Condition on reaction time varies among players.
Interpretation 8: This model tells you how Phrase_Condition and Game_Version interact to influence reaction time, and how this interaction effect varies among players.
Model 6 vs. Model 7:
Model 6: Simpler, assumes the effect of Phrase_Condition is consistent across players. Model 7: More complex, allows the effect of Phrase_Condition to vary by player, providing a more individualized understanding of how Phrase_Condition impacts reaction time.
- Model Fit:
- AIC and BIC: Both AIC and BIC are lower for Model 7 compared to Model 6, indicating that Model 7 provides a better fit to the data despite having more parameters. Lower values of AIC and BIC suggest a model with a better balance of goodness-of-fit and complexity.
- Log-Likelihood: Model 7 has a higher log-likelihood (less negative) compared to Model 6, which also indicates a better fit.
- Deviance:
- The deviance is lower for Model 7 (17477) compared to Model 6
(18753), indicating that Model 7 explains more of the variability in
Reaction_Time.
- The deviance is lower for Model 7 (17477) compared to Model 6
(18753), indicating that Model 7 explains more of the variability in
- Chi-Square Test:
- The Chi-Square test for the difference in deviance between the models is highly significant (Chi-Square = 1275.9, p < 2.2e-16). This significant result indicates that Model 7 provides a significantly better fit to the data than Model 6.
- The degrees of freedom for the test is 9, corresponding to the difference in the number of parameters between the two models (27 - 18).
What Model 7 Tells You:
- Improved Fit: Model 7, with random slopes for
Phrase_ConditionbyPlayer_ID, fits the data significantly better than Model 6. This suggests that allowing the effect ofPhrase_Conditionto vary across players captures important variability inReaction_Time. - Individual Differences: Model 7 accounts for
individual differences in how
Phrase_Conditionaffects reaction time, which is not captured in Model 6.
Conclusion:
Model 7 is a better choice for your data as it provides a
significantly improved fit by allowing for random slopes for
Phrase_Condition. This model acknowledges that the effect
of Phrase_Condition on Reaction_Time varies
among players, capturing individual differences that Model 6 does
not.
Reformat the data to answer each Accuracy question
# Dataframe for learning (Accuracy over Time)
data_learning <- data %>%
group_by(Player_ID, Question_Num) %>%
summarise(Average_Accuracy = mean(Answer), .groups = 'drop')
# Dataframe for hardest condition
data_condition <- data %>%
group_by(Player_ID, Phrase_Condition) %>%
summarise(Average_Accuracy = mean(Answer), .groups = 'drop')
# Dataframe for the impact of different versions
data_version <- data %>%
group_by(Player_ID, Game_Version) %>%
summarise(Average_Accuracy = mean(Answer), .groups = 'drop')
# Dataframe for rate of improvement depending on the version and condition
data_improvement <- data %>%
group_by(Player_ID, Question_Num, Phrase_Condition, Game_Version) %>%
summarise(Average_Accuracy = mean(Answer), .groups = 'drop')What is the average accuracy marginalizing over everything?
Ob: ac ~ 1 + (1 | participant)
# Dataframe for overall average accuracy per participant
data_ac <- data %>%
group_by(Player_ID) %>%
summarise(Average_Accuracy = mean(Answer), .groups = 'drop')
# Fit the model
model2 <- lm(Average_Accuracy ~ 1, data = data_ac)
# Obtain and print a summary of the model to examine fixed and random effects
print(summary(model2))##
## Call:
## lm(formula = Average_Accuracy ~ 1, data = data_ac)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.114045 -0.035124 0.009193 0.035259 0.089426
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 0.8342 0.0086 97 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.05302 on 37 degrees of freedom#Diagnostic plots
#plot_residuals_diagnostics(model2)
plot(model2)
## hat values (leverages) are all = 0.02631579
## and there are no factor predictors; no plot no. 5
# Diagnostic checks - Simulating residuals to validate model assumptions
residuals_simulation <- simulateResiduals(fittedModel = model2, n = 500)
plot(residuals_simulation)
# Model diagnostics
check_model(model2)
# If assumptions are not met, consider re-fitting the model with transformations or different specifications
# model_revised <- update(model, . ~ . + log(Reaction_Time))
# print(summary(model_revised))How does the effect of condition vary depending on the version?
4b: ac - 1 + condition * version + (1 + condition | participant)
# Dataframe for condition and version interaction
data_condition_version <- data %>%
group_by(Player_ID, Phrase_Condition, Game_Version) %>%
summarise(Average_Accuracy = mean(Answer), .groups = 'drop')
#This doesnt work:
#model4 <- lmer(Average_Accuracy ~ 1 + Phrase_Condition * Game_Version + (1 + Phrase_Condition | Player_ID), data = data_condition_version)
model4 <- lmer(Average_Accuracy ~ 1 + Phrase_Condition * Game_Version + (1 | Player_ID), data = data_condition_version)
# Obtain and print a summary of the model to examine fixed and random effects
print(summary(model4))## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: Average_Accuracy ~ 1 + Phrase_Condition * Game_Version + (1 |
## Player_ID)
## Data: data_condition_version
##
## REML criterion at convergence: -360.5
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -3.5947 -0.3806 0.1822 0.5451 1.9247
##
## Random effects:
## Groups Name Variance Std.Dev.
## Player_ID (Intercept) 0.000692 0.02631
## Residual 0.002653 0.05150
## Number of obs: 152, groups: Player_ID, 38
##
## Fixed effects:
## Estimate Std. Error df t value
## (Intercept) 0.961806 0.019278 120.524055 49.892
## Phrase_Condition2 -0.027637 0.024280 102.000000 -1.138
## Phrase_Condition3 0.002905 0.024280 102.000000 0.120
## Phrase_Condition4 -0.222700 0.024280 102.000000 -9.172
## Game_Version2 -0.003531 0.027263 120.524055 -0.130
## Game_Version3 0.008912 0.026573 120.524055 0.335
## Game_Version4 0.003053 0.026573 120.524055 0.115
## Phrase_Condition2:Game_Version2 -0.021755 0.034336 102.000000 -0.634
## Phrase_Condition3:Game_Version2 -0.080703 0.034336 102.000000 -2.350
## Phrase_Condition4:Game_Version2 -0.012499 0.034336 102.000000 -0.364
## Phrase_Condition2:Game_Version3 -0.029769 0.033467 102.000000 -0.890
## Phrase_Condition3:Game_Version3 -0.066345 0.033467 102.000000 -1.982
## Phrase_Condition4:Game_Version3 -0.068849 0.033467 102.000000 -2.057
## Phrase_Condition2:Game_Version4 -0.025387 0.033467 102.000000 -0.759
## Phrase_Condition3:Game_Version4 -0.018297 0.033467 102.000000 -0.547
## Phrase_Condition4:Game_Version4 -0.046639 0.033467 102.000000 -1.394
## Pr(>|t|)
## (Intercept) < 2e-16 ***
## Phrase_Condition2 0.2577
## Phrase_Condition3 0.9050
## Phrase_Condition4 5.52e-15 ***
## Game_Version2 0.8972
## Game_Version3 0.7379
## Game_Version4 0.9087
## Phrase_Condition2:Game_Version2 0.5278
## Phrase_Condition3:Game_Version2 0.0207 *
## Phrase_Condition4:Game_Version2 0.7166
## Phrase_Condition2:Game_Version3 0.3758
## Phrase_Condition3:Game_Version3 0.0501 .
## Phrase_Condition4:Game_Version3 0.0422 *
## Phrase_Condition2:Game_Version4 0.4499
## Phrase_Condition3:Game_Version4 0.5858
## Phrase_Condition4:Game_Version4 0.1665
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1##
## Correlation matrix not shown by default, as p = 16 > 12.
## Use print(summary(model4), correlation=TRUE) or
## vcov(summary(model4)) if you need it#Diagnostic plots
plot_residuals_diagnostics(model4)
## `geom_smooth()` using formula = 'y ~ x'
# Diagnostic checks - Simulating residuals to validate model assumptions
residuals_simulation <- simulateResiduals(fittedModel = model4, n = 500)
plot(residuals_simulation)
# Model diagnostics
check_model(model4)
# Check for influential cases potentially affecting the model
#influence_measures <- influence(model4)
#plot(influence_measures, which = "cook")
# If assumptions are not met, consider re-fitting the model with transformations or different specifications
# model_revised <- update(model, . ~ . + log(Reaction_Time))
# print(summary(model_revised))
# Model interpretation - examining fixed and random effects
fixed_effects <- fixef(model4)
random_effects <- ranef(model4)
# Printing fixed effects for interpretation
print(fixed_effects)## (Intercept) Phrase_Condition2
## 0.961805556 -0.027637486
## Phrase_Condition3 Phrase_Condition4
## 0.002905329 -0.222699519
## Game_Version2 Game_Version3
## -0.003531444 0.008912037
## Game_Version4 Phrase_Condition2:Game_Version2
## 0.003053124 -0.021755046
## Phrase_Condition3:Game_Version2 Phrase_Condition4:Game_Version2
## -0.080703094 -0.012498578
## Phrase_Condition2:Game_Version3 Phrase_Condition3:Game_Version3
## -0.029769227 -0.066344514
## Phrase_Condition4:Game_Version3 Phrase_Condition2:Game_Version4
## -0.068849222 -0.025387477
## Phrase_Condition3:Game_Version4 Phrase_Condition4:Game_Version4
## -0.018297161 -0.046639106# Printing random effects for interpretation
print(random_effects)## $Player_ID
## (Intercept)
## 1 0.0055429197
## 2 -0.0042927426
## 3 0.0191750711
## 4 0.0080938598
## 5 -0.0040876506
## 6 -0.0209085239
## 7 0.0051523019
## 8 0.0052127448
## 9 -0.0138879801
## 10 0.0015230378
## 11 0.0041825285
## 12 -0.0169229170
## 13 0.0108699280
## 14 0.0094571851
## 15 0.0251881424
## 16 -0.0186290065
## 17 0.0137104974
## 18 -0.0293793958
## 19 -0.0279570021
## 20 -0.0084522419
## 21 -0.0097445849
## 22 0.0112085579
## 23 0.0054259932
## 24 -0.0285942721
## 25 0.0262337031
## 26 0.0053075448
## 27 0.0012614904
## 28 0.0253108116
## 29 0.0287031571
## 30 0.0152586747
## 31 -0.0031152626
## 32 0.0002546655
## 33 0.0096590268
## 34 0.0101938355
## 35 0.0180955406
## 36 0.0031894399
## 37 -0.0444359890
## 38 -0.0378030886
##
## with conditional variances for "Player_ID"# Conducting pairwise comparisons of phrase conditions using estimated marginal means
emm_results <- emmeans(model4, specs = pairwise ~ Phrase_Condition)## NOTE: Results may be misleading due to involvement in interactionsprint(summary(emm_results))## $emmeans
## Phrase_Condition emmean SE df lower.CL upper.CL
## 1 0.964 0.00939 121 0.945 0.983
## 2 0.917 0.00939 121 0.898 0.936
## 3 0.925 0.00939 121 0.907 0.944
## 4 0.709 0.00939 121 0.691 0.728
##
## Results are averaged over the levels of: Game_Version
## Degrees-of-freedom method: kenward-roger
## Confidence level used: 0.95
##
## $contrasts
## contrast estimate SE df t.ratio p.value
## Phrase_Condition1 - Phrase_Condition2 0.04687 0.0118 102 3.961 0.0008
## Phrase_Condition1 - Phrase_Condition3 0.03843 0.0118 102 3.248 0.0084
## Phrase_Condition1 - Phrase_Condition4 0.25470 0.0118 102 21.525 <.0001
## Phrase_Condition2 - Phrase_Condition3 -0.00843 0.0118 102 -0.713 0.8918
## Phrase_Condition2 - Phrase_Condition4 0.20783 0.0118 102 17.565 <.0001
## Phrase_Condition3 - Phrase_Condition4 0.21627 0.0118 102 18.277 <.0001
##
## Results are averaged over the levels of: Game_Version
## Degrees-of-freedom method: kenward-roger
## P value adjustment: tukey method for comparing a family of 4 estimates